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ABSTRACT 

One of the missions of the Research on Ev&IClviation 
Program for 1983 , is to increase the capacity of the Nottb^west 
Regional Educational Laboratory (NWREL) to provide region^lQ. services 
in the ireas of evaluation* policy analysis, and especiall w^ cost 
analysis, T^o provide this service, the program needed sgetftiific + 
information on regional service needs and tKa eKistin^r mkiiaXm and 
support neacds of NWREL staff* Three major sources of .jyntotfiriMnation were 
used in ds^eessing these heeds* The first was a recently c^rampletad 
program stuc=3y of the regional effects of the Education Con^sol idat ion 
and IniprovaCTent Act on evaluation practice at the state aftdbd local 
levels* The second source of information was a needs sans imng survey 
conducted fr^y the NWREL Training Center* which included a gtj-tudy of 
local educatsbion agency superintendents 9 needs regarding in-service 
practices/ it3*i teres ts , The third source of information f eWMd on NWREL 
staff perceptions of their own needs in the area of cost tnae'thods 
based on th^^ types of service they have provided, or antidiiipate 
providing, Ao clients* The results of these needs aiiaiUnimit t 
activities Bnd their implications for training and technieasal 
assistance Sn the areas of cost analysis , policy analysis , and other 
evaluation methods are the focus of this report, (■] 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation me thodologies for 
use in education. This document is one o£ a series of paper s and 
reports produced by program staff, visiting scholars, adjunct 
scholars, and project c:ollaboratots--aii members of a cooperative 
network of colleagues working on the development of new 
methodolog las • 

How can the Research on Evaluatk i Program improve Laboratory 
capacity to provide increased regional services in the areas of 
cost analysis, policy analysis, and other evaluation methods? 
What regional needs in these areas can be anticipated and what do 
Laboratory staff need in order f;o improve their field service? 
This report discusses several courses of information gathered by 
the program to shape its capacity building efforts for 1983* 

Nick L* Smith. Editor 
Paper and Resort Series 
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NEEDS ASSESSMENT SUMMARY t 
COST ANALYSIS , POLICY ANALYSIS t AMD 
OTHER EVALUATION ffilTHQDS 

One off the missions of the Research on Evaluation Program 
(ROEP) for 1983, is to increase the capacity of the Northwest 
Regional Educational Laboratory (NWREL) to provide regional 
services in the areas off evaluation, policy analysis, and 
especially cost analysis, Since the ROE? is a research rather 
than service focused program and because it does not have a 
sufficiently large staff to provide field services in these 
areas, the program is working collaboratively to aid other NWREL 
programs in developing this field service capacity* 

In order to provide this capacity building support, the 
program needed specific information on regional service needs and 
the existing skills and support needs of NWREL staff. Three 
major sources of information were used in assessing- these needs. 
The first was a recently completed program study of the regional 
effects of the Education Consolidation and Improvement Act (EC I A) 
on evaluation practice at the state and local -levels. The second 
source of information was. a needs sensing survey conducted* by the 
NWREL Training Center, which included a study of local education 
.agency (LEA) superintendents 1 needs regarding inser vice 
practices/interests. The third source of information focused on 
NWREL staff perceptions of their own needs in the area of cost 
methods based on the types of service they have provided, or 
anticipate providing, to clients. 

The results. of these needs assessment activities and their 
implications for ROEP support of training and technical 
assistance in the areas of cost analysis, policy analysis, and 
other evaluation methods is the focus of this report- 



ECIA Study 

The ECIA study was a ten-month longitudinal study of the 
effects of the Education Consolidation and Improvement Act of 
1981 (ECIA) on ©valuators ,in five western state departments of 
education and five large school districts* The study is Report 
No, 79 of the ROEP Paper and Report Series, 

A semi-structured interview instrument was developed and 
revised on the basis of pilot trials* The form was modified as 
necessary to reflect changing field conditions, However, the 
same basic questions were retained throughout the study, and they 
formed the basis of the results presented here. Respondents were 
interviewed periodically between January 1982 and November 1982 
to chart changing responses to^ the new legislation* 

Over 30 professional educators in five western states 
(California, Montana, Oregon, Utah, and Washington) were 
interviewed during the course of the study, In November the 
final interview respondents numbered 2 4 and were classified as 
follows;* 

5 SEA evaluation directors 
5 LEA evaluation directors 
5 SEA Chapter *2 ©valuators 
5 LEA Chapter 1 ©valuators 
4 LEA Chapter 2 sources 

Overview of Findings 

ECIA has two major parts. One is Chapter 1, which is 
primarily a "reiteration of Title I of the Elementary and 
Secondary Education Act (ESEA) . Chapter 1 deals with 
compensatory education of economically and educationally 
disadvantaged children. The second part of ECIA is a block grant 
which consolidates 30-40 previously categorical programs* These 
include basic skills education m and gifted education programs, as 
well as library and instructional materials and equipment 
grants, In light of these two major parts of the legislation, 
the findings of the ECIA study are grouped under two categories. 
Chapter 1 and Chapter 2, 
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Summary of Preliminary Chapter 1 Finding?. Both state 
education agency (SEA) and local, education agency (LEA) , 
evaluators reported that Title I paperwork was not seen as 
burdensome and unnecessary, nor was any major change in this 
regard reported at the local level relative to Chapter 1 
paperwork. In addition, overall* state and local Chapter 1 
directors reported little perceived change in administrative or 
evaluation flexibility, decentralization of responsibility, or 
administrative burden regarding rules and regulations, A major 
reason for the perceived continuity in administrative practice is 
that the formal Title I Evaluation and Reporting System (TIERS) 
is being retained in all states represented in this study. The 
changes that .have occurred concern dropping the collection of 
certain types of data (e,g,, ethnic data, parent council data, 
staff training data, and project data)., a point specifically 
addressed in the new legislation, 

LEA Chapter 1 directors did, however, report a perceived 
relaxation in the testing and^reporting requirements of 
Chapter 1, For example, one district is contemplating using a 
criterion referenced testing program* Previously, such programs 
were not encouraged since it is difficult to generate student 
pre- and post-test data which can be relied upon to give as 
accurate "a picture of growth as scores from standardized,, 
norm-referenced tests. In another district, it was said that 
percentile scores would be emphasized instead of the Normal Curve 
Equivalent (NOT) scores favored by TIERS • Both of these changes 
were perceived by local directors as efforts to make, testing and 
test results more meaningful to local audiences, 
«■ An increased emphasis on sustained effects studies was 
.reported at both the state and local levels, consistent with the 
emphasis on such studies in the legislation* Sustained effects 
studies trace the impact of a program over time* The impact may 
be defined in terms of student achievement or some ojher program 
characteristic of interest, Such studies can be very useful in- 
monitoring program quality, planning program improvement, and 
reporting program impact. Sustained effects studies represent 



another way of making evaluations/relevant to local audiences 
since they focus on questions of interest to local decision 
makers, including administrators* teachers* parents* and students, 

Parent involvement in Chapter 1 programs is taking on new 
forms consistent with the change in emphasis in the legislation, 
For example, in one school district "willingness to work* was 
being valued over "representativeness" in forming the school 
parent councils. In another district the new Chapter 2 advisory 
council will include parents from the Chapter 1 council resulting 
in more efficient administration of the two programs* , 

A clear message regarding, Chapter 1 came from our respondents. 
As a continuation of a well institutionalised program, Chapter 1/ 
Title I evaluators will experience few immediate changes. 
However, some of the testing and administrative changes could 
.accumulate to produce an impact iu the future, possibly resulting 
in less nationwide data available for Congress to examine the 
overall effects of Chapter 1. 

Chaptei; 1 budget reductions have had a more dramatic impact 
on the local level than the new legislation, resulting in cuts in 
instructional staff, and consequently, a reduction in the number 
of students served, In discussing the impact of losing almost 
one million dollars ~b£ Chapter 1 "monies, a school district 
e valuator noted, 

The real change has come about because of decreased 
' funding, which has severely crippled our program, I was 
at a school this morning that had five aides and a 
teacher last year; this year it has one aide and a 
half-time teacher. That's a t ^_*. difference in numbers of 
kids they can serve, etc. They've reduced their math 
services to working with 10 kids as opposed to 40, That 
resulted because of budgetary cuts, not Chapter 1 
regulation. 

This district budget reduction had lowered by 1000 the number of 
children served. As the respondent noted, "when you go to 
individual schools you really see what that means, " 

The most notable immediate change regarding Chapter 1, then, 
is in regard to the level, of service being offered. The 
reduction in staff, in the number of students served, and perhaps 



in the quality of that service , will have long range affects on 
Chapter 1 impact and on the overall effectiveness of compensatory 

education, ^ 

* 

Summary of preliminary Chapter 2 Findings , Most states 
reported that their Chapter 2 evaluations will consist of 
jp'^n i tor i ng Chapter 2 expenditures for materials using simple cost 
accounting procedures. *At least initially, these states will 
evaluate expenditures for programs in terms of the number off 
students served and staff involved, This concern with the . . 
numbers involved in programs is to a great extent a reflection of 
the fact that, typically, 80 percent or more of Chapter 2 funds 
will go for instructional materials and equipment (mostly 
microcomputer hardware) * The existing categorical programs are 
likely to take the remaining Chapter 2 funds. 

SEA respondents were well aware of the legislative mandate to 
conduct Chapter 2 evaluation studies beginning in fiscal year 
1984, and are communicating -Jthis to local administFators, Some' 
LEA evaluatipn directors are considering innovative research and 
evaluation activities under Chapter >2 including! 



- the study of school practices and learning using 
ethnographic approaches, 

~ the study of teaching behaviors and putcomes using 
causal modeling methods, 

- the adaptation of the evaluation components of 
previous categorica'l programs,- \* * s ' 

* 

- the development of lodally run mini-grant programs. 3 t . 

■ \ 

Similar non-traditional efforts may emerge in other districts. 

Most larg% school districts in our study reported a^r^duction 
in funds under Chapter 2, in comparison with the funds they 
received under the previous categorical aid programs. As a 
result, large school district respondents reported the need to ^ 
find ways tp 4i select among programs competing for reduced ^funds./ 
Some* small school districts have had modest increases in funding 
as a result of Chapter 2 allocations, typically increases of $800 
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to $2500 per district, and are seeking ways to join with other 
districts to get the most from their increases. 

Almost all the SEA and LEA Chapter 2 respondents reported 
increased, flexibility and depentraiization a$ the local level as 
a result of » the Chapter 2 legislation. They had mixed feelings 
about possible reductions in administrative burden, however. 
There is also considerable uncertainty about future evaluation 
activities at 1 the local level under Chapter 2. Once programming 
decisions have been made and the first round of materials and 
microcomputer equipment purchases completed, it is expected that 
©valuators will move from monitoring expenditures to assessing 
the quality and impact of Chapter 2 programs, 




ervice Sur vey* 



A survey designed to gather information about inservice 
programs throughout the region was mailed to half of the region's 
superintendents during June 198 2, A total of 112 surveys were 
returned. A summary of the responses is presented here, In most 
categories, the responses total more than 100 percent because 
respondents checked more than one Response. . - . ... .. . 

Organization of Inservice Programs 

• Forty-six percent (46%j of the survey respondents 
indicated tnat their district had a district-wide 
inservice program* 

• Twenty-nine percent (29%) said there were inservice 
courses available from the district, 

• Forty-seven percent (47%) said the district will 
contract for inservice courses at the request of the 
building staff. 



* The information in this section was provided by Judy Bridges, 
Division of Planning and Service Coordination, NWKEL, in a 
memorandum dated August 23, 1982. 



© Thirty percent (301) said building principals are 
responsible for conducting inservice, 

m One-half (50%) of the districts responding required 

inservice for all staff, while forty-seven percent 47% 
of the districts make inservice available but not 
required , 

Comments regarding the organisation of district inservice 
programs focused on using intermediate, agencies to provide 
inservice. 

Funding of Pi strlctlin service -Programs 

There was clear consensus on this question* Eighty-three 
percent (83%) of the respondents said that funding is set c;t the 
district level. 

Of the remaining answers* 14 percent said funds were 
allocated at the building level, 7 percent said the amount was 
specified on a per-teacher basis* 

Optimum Inservice Training Setting ^ 

Overall, 74 percent of the rer^ondents indicated they would 
support their staff attending training sessions at NWREL. There 
was no clear preference for the best time. Forty-eight percent 
(48%) said" during school time was best, while 36 percent (36%) 
said Saturdays, and twenty-eight percent (28^) said^evenings, 
This totals more than 100 percent because people selected more i 
than one answer. 

In general, the majority of respondents felt full-day or 
multiple sessions -were- the-opt-imum— le nqth Many indica^ri it- 
depended on the type of session and the content, A total of 
sixty-nine percent (69%) wanted inservice sessions to carry 
university credit and fifty-three percent (53%) wanted release 
time for teachers. 

Overall, the answers to the survey questions were diverse. 
NG clear portrait of regional inservice programs emerged. 
However, a prioritized listing of topics of interest to those 
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responding will serve staff well in meeting the needs of the 
region* That list follows ; 



Computer Literacy for Administrators 

How to Increase Learning Time 

Program Evaluation for Busy Administrators 

Evaluation Microcomputer Software 

Instuctional Management 

Teacher Performance Evaluation for Staff Development 
Diagnosing instructional Needs 
Instructional Leadership Training 
Effective Schooling 

Using Writing Assessment to Teach Writing Skills 
Developing an Action Plan for School/Community Relations 
Teamwork and Teambuilding it r Administrators and 

Management Team 
How to Plan and Conduct a Needs Assessment 
Parent Involvement „ 
Community Involvement 

Assessing Vocational Education Effectiveness at the 

Secondary Level 
Conflict Resolution workshop 
Assessing Communication Skills 
School Finance for School Administrators 
interpersonal Communications 
Working with Community Advisory Committees 
Functional Level Testing 

Developing Bid Specs for Hardware and Software 
Diagnosing Your Classroom's Multicultural Needs 
Test Planning 

Intercultural Communications Skills 
Guidelines for Producing Films and video 
Anti-Discrimination Law 

The Teacher , The Classroom and Multicultural Education 

In summary, if we consider the top five topics on the above 
list, we see microcomputers appearing twice* They are first in 
the topic of general literacy which should include an 
introduction to policy issues and to hardware selection* Second, 
they appear as part of the topic regarding" software-evaluation* 
In addition, another oC the five top topics concerns evaluation, 
namely, Program Evaluation for Busy Administrators, This topic 
may well be related to administrators 1 concerns over evaluating 
the new Chapter .2 programs, and over general issues of 
accountability. In any case, the results from this survey have 
direct implications for ROEP support 01 training and technical 
assistance* 
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Cost Analysis Survey 



On February 8, 1983, Dr. Randall Eberts from the University 
of Oregon Economics department led a workshop on cost analysis 
for the staff of the Northwest Regional Educational Laboratory 
(NWREL) , To prepare for that workshop, and to get a clear idea 
of the needs of NWREL staff regarding cost analysis, a needs 
assessment instrument was administered to those staff members who 
had been doing cost studies and/or were interested in attending 
Dr* Eberts 1 workshop, (A copy of the needs assessment instrument 
and a summary of the results from Or* Eberts' perspective are 
included as Appendix A of this report.) 

There were 20 respondents to the needs assessment 
questionnaire. Those who completed it represented four NWREL 
divisions. Over 80 percent of the respondents were from the 
Evaluation and Assessment Division* There was one person from 
each of the following divisions: Multicultural Education, 
Instructional Improvement, and Planning and Service Coordination, 

To understand the kinds of cost—related problems that NSREL 
staff deal with, respondents were asked to describe the cost 
studies they had worked on or anticipated working on, up to a 
maximum of three studies. Fifty percent of the respondents 
reported one study. Thirty percent reported two or three 
studies, and only four out of the twenty (20 percent) did not 
report any studies. 

Summary of Results; Kinds of Studies 

The results of the needs assessment survey reveal four 
different groups of^cos^—related-^studies, -— The cost— analysis ..... 
problems most frequently confronting NWHEL staff involve 
comparing the costs of such entities asi 

• four-day versus five-day school week, 

# alternative early childhood, reading, Chapter 1, and 
other educational programs, 

m use of published, personally developed, or 

performance tests for measuring student growth, 

9 
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© alternative instructional materials, 
m various instructional settings for children of low 

income families, 
m alternative vocational education centers* 

Studies of such comparisons tend to focus only on costs and 
not the relative outcomes of alternatives. 

Another set of studies deals with cost feasibility. They 
focus on such topics as: 

m student activity programs in seven 
m ownership versus contracting, or a 

bus service, 
m test item banking alternatives, 
& competing curricula. 

.-While the first two groups of topics relate to school 

district issues, a third group of studies is aimed at answering 
questions relevant to NWREL. These include: 

m the costs of establishing technical Assistance Center 

field offices, 
m the costs of storing computer literacy test item 

banks, 

m the cost of buying computer time versus purchasing a 
computer. 

A final set of studies involves cost questions not 
specifically related to NWREL or school districts; for 
examplei 

m a four-site study off the costs related to various 
teacher preparation program configurations, 

9 the development of a model for assessing the cost 
effectiveness of adult functional literacy training 
programs , 

* a cost-benefit analysis for the Philippine Ministry 
oc Education and Culture regarding a new national 
English curriculum, 

• the development of a statewide resource allocation 
procedure. 

"Clearly, NWREL-staf f- are confronted with a wide range of 
cost-related problems. 

Summary of Results: Strength and Needs 

The needs assessment questionnaire also contained five 
questions designed to elicit staff members 1 perceptions of 
s^fcreng-ths_and_n eeds in regard to cost-analysis studies. The 
.10 



different schools, 
combination, for 
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first question asked, "What are the typical cost-analysis related 
policy , practice , and/or outcome questions which you encounter or 
anticipate encountering in your work?* 1 When content areas are 
ignored, the responses to this question were very consistent* 
The most frequent response was that cost analysis was usually 
encountered as part of normally conducted program evaluations 
(n 7) , and more specifically, when comparing two programs or 
alternatives (n = 7) , Four individuals reported that their most 
frequent task was simply to identify costs related to programs, 
and one person identified budgeting issues* Finally, four others 
reported that cost analysis was frequently encountered as a 
result of cutbacks which necessitated financial justification for 
program continuation. 

The second question asked, "What problems have you 
experienced, or do you anticipate experiencing, in relation to 
these (cost=analysis) questions? " For most respondents, the 
predominant problems seemed to result from lack of experience 
and/or knowledge about cost analysis methodologies, Three people 
commented very directly that dealing with eost^analysis at any 
level is a problem for them, Four people reported that their 
inexperience with the analyses makes it difficult for them to 
know which cost methodology is appropriate in any given 
situation. Others* however* were more articulate in identifying 
their problem areas. Identifying inputs or costs (n = 5) and 
assigning values to program outputs (n - 5) were reportedly 
difficult. Two individuals wanted. more understanding about the 
differences between hidden and opportunity costs and three others 
did not know how to value costs in terms of present versus future 
values. Other problem areas included unfamiliar ity with the 
literature in the area (n * 1} , budgeting concerns (n - 1) , and 
the client's inability __to expr" n cos t ne eds (n = 1) , Finally, 
two people expressed frustration at the inequities of political 
pressures versus cost analysis data in influencing final 
administrative decisions, 
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In order to better understand the level of sophistication of 
the workshop audience, the question was asked: "What specific 
knowledge, skills, and experience do you bring to bear on a 
cost-analysis question? R As might be expected* the most 
frequently reported strength was in terms of research and 
evaluation training and experience (n ^ 10) , Four individuals 
cited the prior cos t-ana lysis workshop sponsored by the Research 
on Evaluation program as their primary experience with cost 
analysis* Three others claimed to possess an elementary 
understanding of the various cost analyses (e,g,, cost-benefit 
versus cost-f sasibility) t and two said that their exposure had 
been through "reading" * Only two individuals said that they had 
had prior "hands-on" experience with cost analysis. This is 
interesting , given that 23 cases were described by individuals 
when asked about past, present, or future cost study 
experiences. Evidently, che majority of the project descriptions 
were "anticipated". Six individuals identified more specialized 
or technical backgrounds which they felt may facilitate their 
ability to conduct cost studies, Two of these individuals 
reported accounting backgrounds; two, economics backgrounds; one , 
computer analysis background; and one, an industrial engineer i ny 
background. Finally, three respondents said they had no skills, 
knowledge, or experience with cost analysis. 

Respondents Were also asked to indicate "What knowledge and 
skills would you like to develop in regard to the topic of cost 
analysis?" The most frequent response could be categorized as "I 
want to learn how and when to conduct a cost analysis" (n =_ 11) * 
However, this simplistic response was often qualified by a more 
defined need, such as learning how to identify costs associated 
with one or more programs (n m 5), how to identify indirect, 
direct, and opportunity costs <n = 2) , and how to communicate 
"cost-"r~elated-i-nformati-em (n = S) This__latter response category 
included being able to discuss the possibility of cost analysis 
with clients, or with consultants hired to help conduct the 
analysis; it also included how to communicate cost-analysis 
results in an understandable manner, Finally, assistance in 
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learning how to control for intervening variables (n ^ 1} , how to 
identify program outcomes (n = 1) , and how to operational ize 
alternatives (n - 1) was requested* 

The last question asked, *What kind of support would you need 
in order to confidently pursue cost-analysis questions?" Many 
pe^le stated clearly that they wanted an available consultant 
who could critique and/or direct cost analysis studies (n = 8) * 
Six individuals felt that they needed additional training in the 
area,- and five wanted information on relevant readings in the 
area of coat analysis. Finally, one person said that a/he wanted 
more time in a day in order to be able to learn about cost 
analysis and complete the day's work, and two said they did not 
know how to answer she question * 

To summarize, respondents saw cost analysis as a necessary 
componen t of their program evaluation activities* The ability to 
provide data about program cost-effectiveness to decision makers 
was seen as particularly important in light off the present trend 
towari program cutbacks* In terms of actually conducting a cost 
analysis, there were few individuals who seemed to feel 
comfortable with the prospect of such a task, and even fewer who 
had actually worked on a cost analysis* However, this overall 
lack off knowledge, skill, and experience, coupled with the 
perceived importance^ of cost analysis, contributed to a general 
readiness for learning. Respondents clearly wanted to learn 
whatever was necessary to do cost analysis themselves, Finally, 
perhaps in the event that they did not learn quite enough, the 
need for a consultant to assist in cost analysis studies was 
repot ted , 



Implications for Training 
and Technical Assistance. 



Each of the three studies just reported contains a number of 
findings* We believe that the kinds off training and technical 
assistance support and service most consistent with previous ROEP 
development can be grouped under three categories t cost 
analysis, policy analysis , and other evaluation methods. The 
following discussions focus on (1) these three areas and their 
importance vis a vis the three studies, and (2) the nature of 
potential ROEP support* 

Cost Analysis 

The one constant theme from study to study, state to state, 
and SEA to LEA, was concern over reduced budgets and the 
subsequent effect on staffing and program quality. During a 
period of reductions, cost factors become especially important, 
In this regard, local and state ©valuators will need training and 
technical assistance in the following cost^analysis skills: 

(1) determining the true costs of educational programs, 

(2) identifying the quantifiable effects of programs, <3) placing 
subjective utility, or where possible, objective benefit values, 
on program outcomes, and (4) selecting among various alternatives 
based on their cost implications. 

It is evident from all three studies that state and local 
educators are faced with making choices among instructional 
options , These instructional options include, for example, 
differentiated staffing with aides representing a significant 
proportion of the staff, the establishment of a microcomputer^ 
assisted instructional laboratory, and typical teacher^centered 
instruction. These choices involve costs as well as other 
considerations (e.g., policy, evaluation methods). In regard to 
cost issues, much has been said about the need to identify 
high-outcome/low-cost Chapter 1 programs, However, without 
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training and technical assistance in the four cos t=analysis 
skills noted above, it is unlikely that ©valuators will be able 
to make such identification. 

Host large districts experienced an overall decrease in 
Chapter 2 funds relative to the amount of money available for the 
previous categorical grants. It has been reported that a good 
deal of that money is going toward the purchase of instructional 
materials and equipment (i.e*, microcomputers)* Competition for 
the money remaining after materials and equipment allocations are 
made is especially intense. Previously existing programs* which 
have an established staff and institutional and political 
support* are in the best position to secure continued funding* 
However, previous levels cannot typically be maintained , and new 
projects will continue to vie for resources* 

Cost-analysis methods are one source of information for 
making resource allocation decisions. Large district evaluators 
may pursue cost-related questions, such as whether one program is 
more cost feasible than another* Smaller districts may want to 
determine the feasibility of sharing resources with neighboring 
districts- Knowing when and how to do such studies will take 
considerable training on the part of evaluators. 

As indicated in the Inservioe Survey, the development or "Bid 
Specs for (microcomputer) Hardware and Software 14 is a cost- 
related question of interest to the region's superintendents, 
ROEP can provide support and training for NWREL staff who work 
with clients on cost problems, and can provide a limited amount 
of direct technical assistance to SEA and LEA clients. 

Cost issues and the analysis of costs are not the only 
considerations in determining the overall pattern of programs to 
be funded at the state or local level* Policy analysis is a way 
to integrate cost concerns with the whole constellation of issues 
which revolve around major programming and other major decisions* 
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Policy Analysis 

Policy analysis, or the systematic consideration of issues 
and formulation of guidelines, has the potential for assisting 
state and local ©valuators respond to the complex change issues 
confronting them. That is, consideration of such topics as 
increasing learning time; teacher performance evaluation, 
parent/community involvement, effective schooling, and vocational 
education effectiveness can be facilitated by the use of policy 
analysis. Many of the Inservice Survey topics would benefit from 
the application of policy analysis procedures. 

Chapter 1 policy issues, for example, concern such diverse 
topics as (1) flexibility in evaluation design, data collection, 
and reporting; (2) the reduction of administrative paperwork and 
the general burden of administering programs; and (3) the 
staffing of programs relative to various instructional options 
(e.g., differentiated staffing, microcomputer laboratory, 
traditional teacher-centered instruction) * Flexibility in 
evaluation design, data collection, and reporting can raise 
concerns about the quality of data and its eventual aggregation 
at state and national levels- 
Changes in paperwork and general administrative burden, 
especially if the changes involve reductions, have considerable 
short-term appeal. However, such changes raise issues of 
long-term accountability. For example, program auditors need a 
paper trail to verify proper conduct over a considerable period 
of time- In addition, lessening administrative burden can result 
in decreased supervision and control of program activities, This 
has the potential for enhancing creativity and flexibility, But 
it may also result in a lack of consistent direction and a 
subsequent decrease of overall program quality. Policy analysis 
can facilitate the process whereby the trade-offs inherent in 
such actions are considered and a thoughtful solution is 
formulated. 

Decisions regarding staffing and, more basically, the funding 
of programs, can affect people inside school districts 
(administrators, teachers, students, support staff) and outside 
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(parents and other constituents) * The policy-setting process 
used to guide such decisions must include mechanisms for 
including the opinions of these diverse interest groups* A 
systematic process of policy formulation which begins with 
problem clarification and ends with the development of policy 
guidelines should include parent/community involvement at some 
point or other, As the Inservioe Survey shows, these are topics 
of interest to some superintendents* 

There are also some important equity issues regarding the 
allocation of funds for microcomputers* For example, there is 
growing concern over the equal access to microcomputers by girls 
vs. boys, black and other minority students vs* white students, 
and children in less wealthy schools vs* those in more wealthy 
schools* NWREI* staff could support school administrators through 
training and technical assistance to help them unravel the policy 
issues regarding microcomputer usage and to help them formulate 
coherent policy statements* A microcomputer policy formulation 
guidebook is a potential NWHEL product which could be of 
assistance to state and local decision makers* 

policy issues pertain to both the allocation of funds and 
subsequent assessment of impact of the money spent. For example, 
while there are no specific regulations for evaluating Chapter 2 
programs, there is a requirement for an annual evaluation of the 
effects of programs assisted under this chapter, beginning with 
fiscal year 1984* The general guideline for such evaluations may 
be set through the policy analysis process, but their design and 
implementation must rely on evaluators having the ability to 
select appropriate methods for given evaluation problems, 

B valuation .Methodology 

The changes reported in the EC1A study regarding Chapter 1 
evaluation suggest that there will be more independence and, 
hence, responsibility on the part of local evaluators in 
designing both yearly evaluations and sustained effects studies* 
The changes involved in conducting yearly evaluations may present 
problems for state and local evaluators in that they often do not 
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have the skills necessary for selecting appropriate tests and 

interpreting test results* _ Local ©valuators may also have to set 

h 

up internal monitoring systems to document aspects of their 
programs that were previously handled by state monitors* State 
©valuators may have to rethink the statewide process of aggregat- 
ing data if many districts decide to use non-norm=ref erenced 
tests and report scores other than norm curve equivalents. 

Increased emphasis on sustained effects studies mny also 
cause problems for state and loeal ©valuators, For example, when 
developing an evaluation plan, a local evaluator may need 
assistance in identifying key program characteristics and in 
selecting those areas of potential impact most worthy of 
evaluation* In addition, depending on the characteristic (s) 
chosen and evaluation questions posed, special assistance may be 
needed in selecting and/or designing an appropriate evaluation 
strategy* In essence* local Chapter 1 evaluators need to become 
more self-sufficient in selecting and interpreting tests, and in 
designing evaluation studies which meet local needs, 

In some school districts, mini=grants are being funded with 
Chapter 2 money, These grants are distributed to teachers based 
on an application process which, in many cases, requires an 
evaluation plan* These mini^grants and other Chapter 2 programs 
cover a great variety of programs. Therefore, evaluation plans 
are likely to be diverse* tf 

Setting up evaluations which are responsive to these plans 
will necessitate a broad perspective on evaluation, and a mixed 
repertoire of data collection analysis and reporting strategies. 
Therefore, awareness of the range of evaluation methods available 
and guidance in their appropriate selection and application will 
be of paramount importance as evaluators plan for the evaluation 
of Chapter 2 programs in 1984-1985* As "busy administrators," 
the superintendents expressed their interest in evaluation 
methods in response to the Inservice Survey, 

Given the amount of money being spent on microcomputers, the 
evaluation of educational courseware is one area in which state 
and local evaluators may need training and technical assistance. 
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The involvement of teachers and students is a critical element in 
courseware selection* Sy helping administrators become more 
computer literate as suggested by the Inservicy Survey, we can 
enhance their ability to provide leadership in this area, 
However, it is likely that evaluators will need assistance in 
organizing and conducting such evaluations, 

Microcomputers also offer a variety of very powerful tools to 
evaluators- Application programs such as word processing, data 
base management, statistical analysis* electronic spreadsheets, 
graphics, and communication packages can greatly enhance the 
efficiency and quality of an evaluation, However, evaluators 
will need training in using these tools, 

Clearly, with the added emphasis on Chapter 1 sustained 
effects studies, the diversity of Chapter 2 evaluation concerns, 
and the introduction of new microcomputer related areas of study 
and technology, evaluators will have to broaden their repertoire 
of evalution methodologies* 

Based on the results of the EGIA study, the Cost Analysis 
Survey, and the Inservice Survey, it appears that the areas of 
cost analysis, policy analysis, new evaluation methods, and 
microcomputer evaluation are some of the primary areas of need on 
the part of both state and local evaluators* The Research on 
Evaluation Program has over the last several years been 
developing capabilities in these areas. The challenge is to 
tailor the development of support mechanisms for increasing NWREL 
staff capability in providing training and technical assistance 
to meet evaluators 1 needs* 
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Research on Evaluation Program 
Cost-Analysis Needs Assessment 



1, Cost-analysis Questions 

Please briefly state your answers to the following — :ionsi 

a, What are the typical cost-analysis related policy , practice, 
and/or outcome questions which you encounter or anticipate 
encountering in your work? 



b # What problems have you experienced, or do you anticipate 

experiencing, in relation to these questions? 



c* What specific knowledge, skills, and experience do you bring to 
bear on a cost-analysia question? 



d, what knowledge and skills would you like to develop in regard to 
the topic of cost-analysis? 



e# What kind of support would you need in order to confidently 
pursue cost-analysis questions? 
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2. 



Cost-analysis Information 



Please briefly describe one to three cost-analysis cases you have 
worked on or anticipate working on* 

a, Topic i 

b, Setting: 

c* People involved: 
d* Process: 

e- Actual/anticipated outcome: 

a* Topic t 
b* Setting : 

c, People involved: 

d, processi ^ 
e- Actual/anticipated outeomei 
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a. Topici 



b. Setting i 



e* People involved t 



d, Process t 



e. Ac tual/anciclp^ted gutcwiu^ 



Thank you. 



PJGieg 
1/17/83 



TO! Nick Smiths Director 

Research on Evaluation Program 

Fromi Randall W* Eberts 

Ee« Summary of Needs Assessment for Cost Analysis Workshop 

As consultant for the Cost Analysis Workshop at the Lab, I have 
performed two separate duties- The first was to assess the needs of 
the staff who are or will be involved in cost /benefit analysis. The 
second duty was to conduct a seminar which outlines the basic concepts 
of cost analysis and which provides insight into, the steps required in 
performing an actual cost /benefit study* This memo reports on the first 
of the two duties* It summarizes the responses made by staff members to 
the needs assessment questionaire and offers some insights I have gained 
as to the best way to promote this function at the Lab* \ 

Staff members responded to five basic questions concerning their in- 
volvement with Cost/Benefit analysis* 

1) What are the typical cost-analysis related policy* practice* and/ 
or outcome questions which you encounter or anticipate encountering 
in your work? ■ 

2) What problems have you experienced* or do you anticipate experiencing 
in relation to these questions? 

3) What specific knowledge* skills* and experience do you bring to bear 
on a cost-analysis question? 

4) What knowledge and skills would you like to develop in regard to 
the topic of cost-analysis? 

5) What kind of support would you need in order to confidently pursue 
cos t^analysis questions? 

In addition > those staff members who have been involved in cost-analysis 

in the past briefly described the nature of the work that they had performed. 



I will consider the responses to each question separately. 

A, Typical cost-analysis related issues* 

From the responses to this question* it appears that staff members 
have dealt with a wide range of issues. They reported working on issues 
related to Chapter I programs, district financial problems, industrial 
training programs, performance tests, per pupil costs of various in- 
structional settings and vocational education centers. Although these 
diverse issues illustrate- the utility of cost-analysis , it makes it dif- 
ficult to find ground that is common to all concerned. Responses zo the 
next questions reflect this problem. 

B, Problems experienced or anticipated with cost-analysis. 

Six basic problems encountered or anticipated with cost-analysis 
were identified by the respondents. First, concern was expressed about 
the basic methodology, What are the steps required to perform a cost 
study? Second, along the same lines as the first concern, respondents were 
. interested in sources of studies which addressed issues similar to the ones 
they encountered, The third concern was related to comparing cost alter- 
natives. Two issues were expressed: should the consultant suggest alter- 
natives or merely cost out the alternatives suggested by :he client; how 
detailed must one be in capturing all of the costs associated with each al- 
ternative. The fourth and most frequently cited concern was related to out 
comes, not costs. Respondents expressed difficulty identifying outcomes 
such as industrial training programs or vocational education centers. 
Related to the problem of identifying outcomes was the task of linking 



costs to outcomes- Both of these issues illustrate the need for a well* 
developed model of the inputs and outputs of the process under study * 
Finally, there was some concern that since decisions seem to be motivated 
more by political forces than by economic forces* cost analysis may not be 
a suitable framework of analysis* 

e* skills. 

A broad background of skills were represented by the respondents* 
The most frequently cited skill* however t was related to general evaluation 
of education achie%*ement. The rest included experience in cost analysis , 
experimental design > accounting, econometri cs * and educational research 
training. Since most of the work in co ^sis involves some form of 

evaluating educational progress, it appears chat many of the staff members 
who will be conducting these analyses are well^equipped* The fact that 
very few staff members have had much background in cost analysis and 

statistics s on the other hand, presents a fruitful area for further skill 
development* 

B* Development of Skills* 

The respondents* assessment of the skills required for performing 
cos? analysis reflects in many ways a collectively keen intuition about 
the important ingredients of a cost study. Those individua 1 ** who were un- 
familiar with cost analysis asked for a broad overview of the topic. However 
for those staff members who have had some exposure to cost analysis, more 
specific areas of skill development were identified- For example* four 
respondents wanted !, hands=on M experience identifying outcomes and breaking 
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costs. Others were concerned about recognizing whan cost analysis is 
feasible. One interesting aspect of the skills required to perform cost 
analysis was related to educating clients* One respondent expressed the need 
to cMunicate the analysis in a meaningful way to clients; another felt 
that it is important to help practicioners develop cost analysis skills 
at a rudimentary level. Development of both of these skills will heighten 
the effectiveness of this technique, 

E. Support to pursue Cost Analysis- 

When asked "what support would you need to confidently pursue cost 
analysis*" the two overwhelming responses were (1) readings with lots of 
examples and (2) a consultant available to discuss sample applications. 
Both of these suggestions have been implemented by the Research on Evaluation 
Program* The workshop* which I conducted on February 8* provided participants 
with a number of cost studies relating to education, The case studies included 
evaluation of vocational trainings the use of aides in the classroomi the 
cost=ef f ectiveness of federal programs* and a cost analysis of alternative 
instructional modes* In addition to the workshop, I have been available as 
on on-site consultant to staff members at the Lab who are presently engaged in 
cost analysis, A schedule of these consultations appears at the end of this 
report , 

F, Overview and Comments, 

Cost analysis is a complex and difficult task, It is made even more 
difficult when applied to a process like education in which the outcomes 
are difficult to identify and quantify » and in which it is difficult in certain 
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cases ts liAk JLflpM outcomes* ^hin a^y 5^ ttingi how^aver, cost analysis 

requires a c Q ^iEii£ion of three bagitf skills? fenwledge o £ the process under 
study; the Ce^^Ijyas necessary to evj^luici OtiC^ ones and c est inputs; and 
the inc%i±tiOTi ^Adjudgement necessary to baist^ftht rr-Qo ■ gnition of actual 
costs with the costs' that are feasible * to inc loA e in the s - rudy- Only under * 
very special ^cLiWanees * in which fcsn id entice! proves 5 is studied repeatedly, 
can precise gC^plte specified for eygr^ryorie to follow. Si^xice staff members 
at' the ILab a r £ ciittwtly engaged in &o osc studies efvariot^^s sorts* each study 
will con taifi referent aspects and zo\fflp- ^iicatioti^* Ihirefo^ ^ , the best way to 
train staff ^gifcWi eo perform cost ^fl^^alysis ±0 tttrain £ ham in the basic, 
ingredients C f-i t comprise a cost SC«^)* — . Ma^y &talt&A§ = s already have two 
important skills: Ehey understand t^g m educatioMi pfseeas 3 and they are well- 
versed in e^al^a^n techniques* Thui ^ * in gaiie^ai, only t-^wo skills are lacking; 
teowledge of clii^hniques of costing s inputs % ^jd the jud gement of deciding 
when the ptoc^s^been specified gu^ff iciefttiy and the a oats of the inputs have 
been accounted fof properly. Both o£ these rkillitri r^q= -uired to provide 
an accurate bat feasible analysis. £h%me first e^scttifc tA£" ©u§h workshops and 
readings; the stfHid must come chr©ugCsh !f hands^tis" e*per i* ^nce and exr Dgure 
to actual atudie s under the gui^^^nce of - so^one with -.experience. 
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Log of Consults, z ions with 
Star? ^n^bers 
January IE , 1983 



1 :Q0-1 :25 Dean Nsrziger (Division Dir~ ector) 

1:30-1:55 Gary Estgs (Program Directcx) 
Bill Savard 
Steve Nelson 
Randy Demaline 

2:00=2:25 Steve Murray (Associate P£c- gram Director) 
Roy Gabriel 

2:30-2:55 Larry Picus 
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